Fix oom caused by ujson #336
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
li-wu
requested review from
arctan5x,
GordonWang,
jmeixensperger and
lephino
as code owners
GordonWang
previously approved these changes
Oct 31, 2019
li-wu
force-pushed
the
fix-oom-by-ujson
branch
from
0b5cfcc
to
1c5b59d
Compare
li-wu
commented
Oct 31, 2019
|
|
||
| @echo 'Installing test requirements' | ||
| docker exec -i ${EVENTGEN_TEST_IMAGE} /bin/sh -c "pip install --upgrade pip;pip install -r $(shell pwd)/requirements.txt" | ||
| docker exec -i ${EVENTGEN_TEST_IMAGE} /bin/sh -c "pip3 install --upgrade pip;pip3 install -r $(shell pwd)/requirements.txt;pip3 install git+https://github.com/esnme/ultrajson.git" |
There was a problem hiding this comment.
ultrajson/ultrajson#326. We may need to discard the lib ujson since it is not maintained and updated.
jmeixensperger
approved these changes
Oct 31, 2019
arctan5x
approved these changes
Oct 31, 2019
jmeixensperger
added a commit
that referenced
this pull request
Apr 6, 2020
* Fix structlog dependency for app (#280) * zipfile fix (#284) * Fix bug 286 random token replacement (#287) * Fix bug 286 random token replacement * Change perdayvolume generator logic to get random token value replacement * Versioning scheme (#278) * [global] perDayVolume (#288) * exclude global from perDayVolume assignment * Address comment * Fix security vulnerability issue (#289) * Fix custom plugin stale docs (#290) * Server fix (#293) * Flag added * server fix for count and env clean * Fix bug 285 (#297) * Add syslogAddHeader config directive (#296) * Add syslog header to event in syslog mode * timezone setting bugfix #249 * Using multiprocess pool to address the OOM issue (#301) * Using multiprocess pool to address the OOM issue * Fix test case fail * Remove workerQueue unfinished tasks (#302) * Bumped version to 6.5.2 * controller fix (#304) * controller fix * variable assignment fix (#306) * add healthcheck endpoint and ping it every half an hour (#308) * Fix security vulnerability issue reported by GitHub (#309) * make controller also hit redis on healthcheck (#311) * Fixed typo (#313) * update redis version to use new functions (#318) * Update tutorial related docs (#315) * Revert oom fix change (#320) * make circle ci fail when test case fails (#317) * Add disable logging option to fix oom (#323) * Add disable logging option to fix oom * Fix modular input bug * Fixing api server (#322) * fixed multiple attributes * debug * Py3 upgrade (#291) * init commit for python 3 migration * migrate to py3 for test * Fix import issue * Fix test cases * Fix jinja test case * Fix multiprocess issue * Fix a few warnings * Fix test case fail * Fix test case * Fix cluster test cases * Fix set_conf error * Fix shebang * Remove logutils lib from source * Fix splunkstream import error * Remove logutils from requirements.txt * Fix build spl error * Change code block indent * Fix doc anchor issue * Fix app running on Splunk 8.0 issue * Fix xrange issue introduced by develop branch * Fix merge bug * Fix bug * remove logutils * Fix makefile bug * Fix configparser bug * Change filter function * Bump version to 7.0.0 * Fix dateutil package bug * cleanup memory (#326) * cleanup memory * addressing comment * Fixed standalone healthcheck (#328) * Add python2 libs required by perf test case (#331) * make splunk_eventgen as a module (#332) * Fix oom caused by ujson (#336) * Fix oom caused by ujson * Fix test env * add release automation script (#335) * add release automation script * Fix process leak and start/stop 500 issue (#344) * Fix process leak and start/stop 500 issue * Fix test case fail * Fix high system load issue (#345) * Adding an option (#340) * Adding an option * Adding ujson2 for python3 as well * minor improvements * Scp plugin (#343) * intermediate code for scp_plugin * code clean up * Added doc * revert windbag * Cap * added test * revert (#347) * changing name (#349) * Change (#350) * change token type (#351) * Add multithread support (#353) * Add multithread support * Fix typo * Revert coverage version to 4.5.4 (#355) * Fix CI failure cased by jinja new version and update the log config to reduce log in app (#358) * Add missing documentation for negative floats, as introduced in PR#127 (#360) * Bump nokogiri from 1.10.4 to 1.10.8 in /docs (#359) Bumps [nokogiri](https://github.com/sparklemotion/nokogiri) from 1.10.4 to 1.10.8. - [Release notes](https://github.com/sparklemotion/nokogiri/releases) - [Changelog](https://github.com/sparklemotion/nokogiri/blob/master/CHANGELOG.md) - [Commits](sparklemotion/nokogiri@v1.10.4...v1.10.8) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Li Wu <lwu@splunk.com> * Fix jinja plugin bug and test addon sample (#361) * regex stanza bugfix (#365) * prelim changes * fix regex length matching * fix wildcard sample + csv cases * small csv fix, add test case * revert accidental change * Upgraded test instance * add httpevent collector * forgot to update stanza name * add escape for csv matching + test case * add sample, remove stanza * push not working Co-authored-by: tonyl <tonyl@splunk.com> * Improvement/release script (#368) * add PR to master * update title in sent PR Co-authored-by: Li Wu <lwu@splunk.com> * Correct python2/3 issue (#363) * msg must be a bytes like object * msg must be a bytes like object Co-authored-by: Li Wu <lwu@splunk.com> * update eventgen version to 7.1.0 Co-authored-by: Li Wu <lwu@splunk.com> Co-authored-by: Tony Lee <tonyl@splunk.com> Co-authored-by: Patrik Nordlén <patriki@gmail.com> Co-authored-by: Guodong Wang <wanggd04@gmail.com> Co-authored-by: David Wang <Devmalion@users.noreply.github.com> Co-authored-by: Guodong Wang <gwang@splunk.com> Co-authored-by: Erwin Vrolijk <technimad@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ryan Faircloth <35384120+rfaircloth-splunk@users.noreply.github.com>
jmeixensperger
added a commit
that referenced
this pull request
Apr 15, 2020
* Fix structlog dependency for app (#280) * zipfile fix (#284) * Fix bug 286 random token replacement (#287) * Fix bug 286 random token replacement * Change perdayvolume generator logic to get random token value replacement * Versioning scheme (#278) * [global] perDayVolume (#288) * exclude global from perDayVolume assignment * Address comment * Fix security vulnerability issue (#289) * Fix custom plugin stale docs (#290) * Server fix (#293) * Flag added * server fix for count and env clean * Fix bug 285 (#297) * Add syslogAddHeader config directive (#296) * Add syslog header to event in syslog mode * timezone setting bugfix #249 * Using multiprocess pool to address the OOM issue (#301) * Using multiprocess pool to address the OOM issue * Fix test case fail * Remove workerQueue unfinished tasks (#302) * Bumped version to 6.5.2 * controller fix (#304) * controller fix * variable assignment fix (#306) * add healthcheck endpoint and ping it every half an hour (#308) * Fix security vulnerability issue reported by GitHub (#309) * make controller also hit redis on healthcheck (#311) * Fixed typo (#313) * update redis version to use new functions (#318) * Update tutorial related docs (#315) * Revert oom fix change (#320) * make circle ci fail when test case fails (#317) * Add disable logging option to fix oom (#323) * Add disable logging option to fix oom * Fix modular input bug * Fixing api server (#322) * fixed multiple attributes * debug * Py3 upgrade (#291) * init commit for python 3 migration * migrate to py3 for test * Fix import issue * Fix test cases * Fix jinja test case * Fix multiprocess issue * Fix a few warnings * Fix test case fail * Fix test case * Fix cluster test cases * Fix set_conf error * Fix shebang * Remove logutils lib from source * Fix splunkstream import error * Remove logutils from requirements.txt * Fix build spl error * Change code block indent * Fix doc anchor issue * Fix app running on Splunk 8.0 issue * Fix xrange issue introduced by develop branch * Fix merge bug * Fix bug * remove logutils * Fix makefile bug * Fix configparser bug * Change filter function * Bump version to 7.0.0 * Fix dateutil package bug * cleanup memory (#326) * cleanup memory * addressing comment * Fixed standalone healthcheck (#328) * Add python2 libs required by perf test case (#331) * make splunk_eventgen as a module (#332) * Fix oom caused by ujson (#336) * Fix oom caused by ujson * Fix test env * add release automation script (#335) * add release automation script * Fix process leak and start/stop 500 issue (#344) * Fix process leak and start/stop 500 issue * Fix test case fail * Fix high system load issue (#345) * Adding an option (#340) * Adding an option * Adding ujson2 for python3 as well * minor improvements * Scp plugin (#343) * intermediate code for scp_plugin * code clean up * Added doc * revert windbag * Cap * added test * revert (#347) * changing name (#349) * Change (#350) * change token type (#351) * Add multithread support (#353) * Add multithread support * Fix typo * Revert coverage version to 4.5.4 (#355) * Fix CI failure cased by jinja new version and update the log config to reduce log in app (#358) * Add missing documentation for negative floats, as introduced in PR#127 (#360) * Bump nokogiri from 1.10.4 to 1.10.8 in /docs (#359) Bumps [nokogiri](https://github.com/sparklemotion/nokogiri) from 1.10.4 to 1.10.8. - [Release notes](https://github.com/sparklemotion/nokogiri/releases) - [Changelog](https://github.com/sparklemotion/nokogiri/blob/master/CHANGELOG.md) - [Commits](sparklemotion/nokogiri@v1.10.4...v1.10.8) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: Li Wu <lwu@splunk.com> * Fix jinja plugin bug and test addon sample (#361) * regex stanza bugfix (#365) * prelim changes * fix regex length matching * fix wildcard sample + csv cases * small csv fix, add test case * revert accidental change * Upgraded test instance * add httpevent collector * forgot to update stanza name * add escape for csv matching + test case * add sample, remove stanza * push not working Co-authored-by: tonyl <tonyl@splunk.com> * Improvement/release script (#368) * add PR to master * update title in sent PR Co-authored-by: Li Wu <lwu@splunk.com> * Correct python2/3 issue (#363) * msg must be a bytes like object * msg must be a bytes like object Co-authored-by: Li Wu <lwu@splunk.com> * Release eventgen 7.1.0. Merge to develop branch. (#372) * Updating Master (#312) * Fix structlog dependency for app (#280) * zipfile fix (#284) * Fix bug 286 random token replacement (#287) * Fix bug 286 random token replacement * Change perdayvolume generator logic to get random token value replacement * Versioning scheme (#278) * [global] perDayVolume (#288) * exclude global from perDayVolume assignment * Address comment * Fix security vulnerability issue (#289) * Fix custom plugin stale docs (#290) * Server fix (#293) * Flag added * server fix for count and env clean * Fix bug 285 (#297) * Add syslogAddHeader config directive (#296) * Add syslog header to event in syslog mode * timezone setting bugfix #249 * Using multiprocess pool to address the OOM issue (#301) * Using multiprocess pool to address the OOM issue * Fix test case fail * Remove workerQueue unfinished tasks (#302) * Bumped version to 6.5.2 * controller fix (#304) * controller fix * variable assignment fix (#306) * add healthcheck endpoint and ping it every half an hour (#308) * Fix security vulnerability issue reported by GitHub (#309) * make controller also hit redis on healthcheck (#311) * Update Master (#314) * Fix structlog dependency for app (#280) * zipfile fix (#284) * Fix bug 286 random token replacement (#287) * Fix bug 286 random token replacement * Change perdayvolume generator logic to get random token value replacement * Versioning scheme (#278) * [global] perDayVolume (#288) * exclude global from perDayVolume assignment * Address comment * Fix security vulnerability issue (#289) * Fix custom plugin stale docs (#290) * Server fix (#293) * Flag added * server fix for count and env clean * Fix bug 285 (#297) * Add syslogAddHeader config directive (#296) * Add syslog header to event in syslog mode * timezone setting bugfix #249 * Using multiprocess pool to address the OOM issue (#301) * Using multiprocess pool to address the OOM issue * Fix test case fail * Remove workerQueue unfinished tasks (#302) * Bumped version to 6.5.2 * controller fix (#304) * controller fix * variable assignment fix (#306) * add healthcheck endpoint and ping it every half an hour (#308) * Fix security vulnerability issue reported by GitHub (#309) * make controller also hit redis on healthcheck (#311) * Release 7.0.0 (#324) * Fix structlog dependency for app (#280) * zipfile fix (#284) * Fix bug 286 random token replacement (#287) * Fix bug 286 random token replacement * Change perdayvolume generator logic to get random token value replacement * Versioning scheme (#278) * [global] perDayVolume (#288) * exclude global from perDayVolume assignment * Address comment * Fix security vulnerability issue (#289) * Fix custom plugin stale docs (#290) * Server fix (#293) * Flag added * server fix for count and env clean * Fix bug 285 (#297) * Add syslogAddHeader config directive (#296) * Add syslog header to event in syslog mode * timezone setting bugfix #249 * Using multiprocess pool to address the OOM issue (#301) * Using multiprocess pool to address the OOM issue * Fix test case fail * Remove workerQueue unfinished tasks (#302) * Bumped version to 6.5.2 * controller fix (#304) * controller fix * variable assignment fix (#306) * add healthcheck endpoint and ping it every half an hour (#308) * Fix security vulnerability issue reported by GitHub (#309) * make controller also hit redis on healthcheck (#311) * Fixed typo (#313) * update redis version to use new functions (#318) * Update tutorial related docs (#315) * Revert oom fix change (#320) * make circle ci fail when test case fails (#317) * Add disable logging option to fix oom (#323) * Add disable logging option to fix oom * Fix modular input bug * Fixing api server (#322) * fixed multiple attributes * debug * Py3 upgrade (#291) * init commit for python 3 migration * migrate to py3 for test * Fix import issue * Fix test cases * Fix jinja test case * Fix multiprocess issue * Fix a few warnings * Fix test case fail * Fix test case * Fix cluster test cases * Fix set_conf error * Fix shebang * Remove logutils lib from source * Fix splunkstream import error * Remove logutils from requirements.txt * Fix build spl error * Change code block indent * Fix doc anchor issue * Fix app running on Splunk 8.0 issue * Fix xrange issue introduced by develop branch * Fix merge bug * Fix bug * remove logutils * Fix makefile bug * Fix configparser bug * Change filter function * Bump version to 7.0.0 * Release 7.0.0 (#325) * Fix structlog dependency for app (#280) * zipfile fix (#284) * Fix bug 286 random token replacement (#287) * Fix bug 286 random token replacement * Change perdayvolume generator logic to get random token value replacement * Versioning scheme (#278) * [global] perDayVolume (#288) * exclude global from perDayVolume assignment * Address comment * Fix security vulnerability issue (#289) * Fix custom plugin stale docs (#290) * Server fix (#293) * Flag added * server fix for count and env clean * Fix bug 285 (#297) * Add syslogAddHeader config directive (#296) * Add syslog header to event in syslog mode * timezone setting bugfix #249 * Using multiprocess pool to address the OOM issue (#301) * Using multiprocess pool to address the OOM issue * Fix test case fail * Remove workerQueue unfinished tasks (#302) * Bumped version to 6.5.2 * controller fix (#304) * controller fix * variable assignment fix (#306) * add healthcheck endpoint and ping it every half an hour (#308) * Fix security vulnerability issue reported by GitHub (#309) * make controller also hit redis on healthcheck (#311) * Fixed typo (#313) * update redis version to use new functions (#318) * Update tutorial related docs (#315) * Revert oom fix change (#320) * make circle ci fail when test case fails (#317) * Add disable logging option to fix oom (#323) * Add disable logging option to fix oom * Fix modular input bug * Fixing api server (#322) * fixed multiple attributes * debug * Py3 upgrade (#291) * init commit for python 3 migration * migrate to py3 for test * Fix import issue * Fix test cases * Fix jinja test case * Fix multiprocess issue * Fix a few warnings * Fix test case fail * Fix test case * Fix cluster test cases * Fix set_conf error * Fix shebang * Remove logutils lib from source * Fix splunkstream import error * Remove logutils from requirements.txt * Fix build spl error * Change code block indent * Fix doc anchor issue * Fix app running on Splunk 8.0 issue * Fix xrange issue introduced by develop branch * Fix merge bug * Fix bug * remove logutils * Fix makefile bug * Fix configparser bug * Change filter function * Bump version to 7.0.0 * Fix dateutil package bug * update eventgen version to 7.1.0 Co-authored-by: Tony Lee <tonyl@splunk.com> Co-authored-by: Li Wu <lwu@splunk.com> Co-authored-by: Li Wu <lwu@splunk.com> Co-authored-by: Tony Lee <tonyl@splunk.com> Co-authored-by: Jack Meixensperger <jackm@splunk.com> Co-authored-by: Patrik Nordlén <patriki@gmail.com> Co-authored-by: David Wang <Devmalion@users.noreply.github.com> Co-authored-by: Erwin Vrolijk <technimad@users.noreply.github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Ryan Faircloth <35384120+rfaircloth-splunk@users.noreply.github.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
After a bunch of testing on the OOM issue. I found this is caused by the
ujsonpackage which is needed byconduct2for perf testing. In this pr, I removed the installation step forujsonvia git source code and I kicked two round of tests(one standalone and one triggered by perf test case) based on the new built docker image and the OOM issue is gone now.